Introduction to Machine Learning

1. Introduction

In the first two chapters, we introduced machine learning as a complete project rather than as a collection of algorithms. We learned that a successful project begins with a well-defined problem, continues through data understanding and preparation, model development and evaluation, and eventually reaches deployment and monitoring.

This case study puts that lifecycle into practice. We will follow a fictional bank that wants to predict whether a newly approved loan is likely to default within the next 12 months. The purpose is not to build the most sophisticated model possible. Instead, we will focus on the decisions that must be made before, during, and after model training.

2. The Problem: Should the Bank Approve This Loan?

A bank receives thousands of loan applications every month. Historically, most applications were assessed using a mixture of manually defined rules and expert judgment. The bank now wants to use historical data to estimate the probability that a borrower will default on a new loan.

At first glance, the task appears straightforward: collect historical applications, train a classification model, and use it to predict “Default” or “No Default.” In practice, almost every part of that sentence contains a decision that can affect the validity of the final system.

2.1 The Stakeholders

The ML system may involve several stakeholders:

• Loan officers who use predictions while reviewing applications.
• Risk managers who care about portfolio-level default risk.
• Business managers who care about approval rates, revenue, and losses.
• Compliance and audit teams who need decisions to be explainable and reviewable.
• Data scientists and ML engineers who build and maintain the system.

2.2 The Business Objective

The bank does not actually want a model that merely predicts a label. It wants to make better lending decisions while balancing financial risk, customer experience, and operational constraints.

A useful project statement is therefore:

2.3 What Would Count as Success?

The business objective must be translated into measurable criteria. Possible criteria include:

• Reduce the proportion of high-risk loans that are approved.
• Maintain an acceptable approval rate for low-risk applicants.
• Produce reliable probability estimates rather than only hard class labels.
• Maintain acceptable performance across relevant customer groups.
• Provide enough evidence and documentation for human review.

Notice that none of these statements says “maximize accuracy.” Accuracy is a model metric; it is not automatically the business objective.

3. Defining the ML Task

3.1 What Is One Prediction?

Before collecting features, we must define the unit of prediction. In this case, one prediction corresponds to one loan application evaluated at the time the lending decision is made.

This distinction matters. A dataset in which one row represents a customer-month is not equivalent to a dataset in which one row represents a loan application. The meaning of every feature, the splitting strategy, and even the target variable can change when the unit of observation changes.

3.2 Classification or Regression?

The final decision can be expressed as a binary classification problem:

Default within 12 months? → Yes / No

However, a useful model may produce a probability first:

P(Default within 12 months) = 0.17

The bank can then combine that probability with a lending policy or decision rule. This distinction between prediction and decision will become important later in the case.

3.3 Mitchell’s T/E/P Framework

3.4 The Prediction Horizon

The prediction horizon is part of the task definition. “Will the borrower ever default?” is a different problem from “Will the borrower default within 12 months?” A different horizon produces a different target variable and potentially different feature requirements.

4. Understanding the Data

4.1 A Small Sample

The values in this table are illustrative and synthetic. The units and terminology are chosen to resemble a South Asian lending context rather than represent any particular bank's portfolio.

4.2 A Data Dictionary

4.3 Predictive Does Not Mean Valid

A useful feature is not simply one that correlates strongly with the target. A valid feature must also be available at the moment the prediction is made and be obtained in a manner consistent with the intended deployment process.

5. Exploratory Data Analysis

We now inspect the dataset before choosing an algorithm. The objective of EDA is not merely to produce attractive plots. It is to understand the data-generating process well enough to make defensible preprocessing and modeling decisions.

5.1 What Should We Look For?

• Class distribution and imbalance.
• Missing values and patterns of missingness.
• Impossible or suspicious values.
• Outliers and extreme observations.
• Feature distributions and skewness.
• Relationships between features and the target.
• Potential duplicate or related observations.
• Variables that may leak future information.

5.2 The Target Distribution

Suppose the historical dataset contains 100,000 applications, of which only 7,000 eventually default.

A naïve model that predicts “No Default” for everyone would already achieve 93% accuracy. Yet it would identify no risky borrowers. This is our first indication that the metric must reflect the purpose of the system.

5.3 Missing Values and Missingness Patterns

Suppose 2% of monthly income values are missing and 1% of employment information is missing. We cannot automatically decide that missing values should be removed or replaced. We should first investigate whether the missingness is random and whether the fact that a value is missing carries useful information.

Any imputation statistic used for the eventual model must be learned from the training data only, as discussed in Chapter 2.

6. Designing the Data Pipeline

Our earlier lifecycle separates universal data-cleaning operations from transformations that learn parameters from the data. We now apply that rule to this dataset.

6.1 Before the Split

We may perform data-integrity operations such as:

• Removing exact duplicate application records.
• Standardizing obvious formatting inconsistencies.
• Correcting known data-entry errors according to documented business rules.
• Creating features whose inputs are all available at prediction time.

6.2 After the Split

Operations that learn values or structure from the observed distribution belong inside the training pipeline. Examples include:

• Median imputation.
• Scaling.
• Feature selection.
• PCA or other learned dimensionality reduction.

7. Establishing a Baseline

Before training a sophisticated model, we establish a simple baseline. A baseline answers a fundamental question: Is our ML solution actually better than a reasonable simple strategy?

For this problem, a majority-class baseline predicts “No Default” for every application. Its accuracy is 93%, but its recall for the default class is 0%.

This immediately illustrates why a baseline should be evaluated using metrics relevant to the problem rather than by one aggregate number alone.

8. Choosing What to Measure

8.1 The Confusion Matrix

8.2 Different Errors Have Different Costs

A false negative occurs when the model predicts that a borrower is safe but the borrower eventually defaults. This may create a direct financial loss.

A false positive occurs when the model flags a borrower as risky even though the borrower would not default. Depending on the lending policy, this may mean rejecting a profitable customer or imposing less favorable terms.

Therefore, there is no universal answer to the question “Which is worse?” The answer depends on the decision that follows the prediction.

8.3 Metrics We Might Consider

9. Prediction Is Not the Same as Decision

Suppose a model produces the following probability estimates:

If the bank uses a threshold of 0.50, applicants D and E are flagged as high risk. But why 0.50? There is nothing mathematically inevitable about that threshold.

If the cost of a missed default is much larger than the cost of investigating an additional applicant, the bank may choose a lower threshold. If unnecessary interventions are extremely costly, it may choose a higher threshold.

10. Model Development

Only now do we begin comparing algorithms. Later chapters will study these models individually and in mathematical detail. Here, the purpose is to see how they fit into a larger investigation.

10.1 Candidate Models

• Logistic Regression: a strong, interpretable baseline for binary classification.
• Decision Tree: can represent nonlinear decision rules and is easy to inspect.
• Random Forest: an ensemble that can capture nonlinear interactions robustly.
• Gradient Boosting: often highly competitive on structured/tabular data.

10.2 Do Not Use the Test Set to Choose the Winner

The models and their hyperparameters should be compared using the training data and validation strategy. The final test set must remain untouched until the modeling decisions have been finalized.

If we repeatedly inspect the test score and choose whichever model performs best there, the test set is no longer an unbiased final check of generalization.

11. Error Analysis: Where Does the Model Fail?

After selecting a candidate model, we do not stop at its overall metrics. We inspect its mistakes.

11.1 Questions to Ask

• Are errors concentrated among borrowers with very low income?
• Does performance change for different loan purposes?
• Are borderline cases responsible for most errors?
• Does performance differ substantially across customer groups?
• Are there data-quality problems among the misclassified cases?

11.2 Aggregate Performance Can Hide Important Patterns

Imagine a model with a strong overall score but poor recall for one smaller customer segment, such as borrowers from a particular employment or income category. The overall metric may still look excellent. A responsible evaluation therefore examines performance at multiple relevant levels instead of relying on a single number.

12. Probability Quality and Calibration

A lender may need probabilities rather than only rankings. For example, a predicted probability of 0.70 may be interpreted as “about seven out of ten similar cases default,” but such an interpretation is only justified when the probability estimates are reasonably calibrated.

Two models may rank borrowers similarly while producing probabilities with very different reliability. This is one reason why model evaluation should consider the intended use of the output, not only discrimination metrics.

13. Deployment

Assume the bank decides to use the model as one component of its loan review process. The deployed system might receive an application, run the approved preprocessing pipeline, generate a risk estimate, apply the current decision policy, and present the result to the user interface.

13.1 What Can Go Wrong After Deployment?

• The incoming data may have a different distribution from the training data.
• Some features may become unavailable because an upstream system changed.
• Customer behavior may change because economic conditions change.
• The relationship between features and default may change over time.
• A business policy change may make the original target or threshold inappropriate.

14. Monitoring

Deployment is therefore not the end of the ML lifecycle. The system must be monitored.

Monitoring connects the model to the real world. A model that worked well during development can stop being useful when the environment changes.

15. End-to-End View of the Project

The complete project can now be summarized as:

16. What This Case Study Tells Us About Machine Learning

The most important lesson is that model training is only one part of an ML project. The algorithm is chosen within a larger sequence of decisions about the problem, data, evaluation, deployment, and use of predictions.

The same classification algorithm can be appropriate in finance, healthcare, marketing, or another domain, yet the surrounding ML project may be completely different because the data, errors, stakeholders, costs, and operational environment are different.

17. Interactive Examples

Example 1: Is This Feature Valid?

For each variable, decide whether it is valid for a model making a prediction at loan-application time.

Example 2: Is 93% Accuracy Good?

Example 3: Prediction or Decision?

18. Try It Yourself

19. Key Takeaways

1. An ML project begins with a clearly defined decision problem, not an algorithm.
2. The unit of prediction and prediction horizon must be explicit.
3. A feature can be highly predictive and still be invalid because it contains information unavailable at prediction time.
4. EDA is used to understand the data and uncover problems that can invalidate later modeling.
5. Baselines are necessary for judging whether a model provides meaningful improvement.
6. Accuracy can be misleading when classes are imbalanced.
7. Different errors can have different practical costs.
8. A model prediction and the decision based on that prediction are not the same thing.
9. The test set should remain untouched until final evaluation.
10. Deployment and monitoring are part of the ML lifecycle, not optional afterthoughts.

20. Common Pitfalls

1. Starting with the algorithm: Choosing Logistic Regression or Random Forest before defining the prediction task.
2. Using future information: Including variables that would not be available when the prediction is made.
3. Trusting accuracy blindly: Reporting a high score without checking the class distribution and confusion matrix.
4. Tuning on the test set: Using final-test performance to choose models or hyperparameters.
5. Ignoring the decision context: Treating every prediction as equally valuable or equally costly.
6. Stopping at deployment: Assuming that a model that worked on historical data will work forever.

21. Looking Ahead

This case study intentionally introduced several ideas without developing all of their technical details. The remainder of the course will return to these ideas in a more formal way.

The purpose of the case is therefore not to teach all of those techniques now. It is to give us a concrete problem to return to as we learn the tools that solve different parts of it.

Selected References

1. State Bank of Pakistan (2013). Internal Credit Risk Rating System – Retail Portfolio. This guidance describes the use of application and behavioral scorecards in retail credit, including the selection of default drivers, validation of predictive performance, comparison of predicted and actual defaults, and periodic review and updating of scorecards. Read the SBP guidance →
2. State Bank of Pakistan. Credit Information Bureau (eCIB) — Frequently Asked Questions. This resource explains the role of credit information in lending decisions, including total borrower exposure, repayment history, repayment capacity, and the distinction between factual credit information and a credit rating. Read the eCIB FAQs →